Most of the existing automatic emotion annotation methods only extract the single recognition feature from acoustic layer or language layer. While Yi language is affected by the factors such as too many branch dialects and high complexity, so the accuracy of automatic annotation of Yi language with single-layer emotion feature is low. Based on the features such as rich emotional affixes in Yi language, a double-layer feature fusion method was proposed. In the method, the emotional features from acoustic layer and language layer were extracted respectively, the methods of generating sequence and adding units as needed were applied to complete the feature sequence alignment, and the process of automatic emotion annotation was realized through the corresponding feature fusion and automatic annotation algorithm. Taking the speech and text data of Yi language in a poverty alleviation log database as samples, three different classifiers were used for comparative experiments. The results show that the classifier has no obvious effect on the automatic annotation results, and the accuracy of automatic annotation after the fusion of double-layer features is significantly improved, the accuracy is increased from 48.1% of acoustic layer and 34.4% of language layer to 64.2% of double-layer fusion.